Video processing device for generating time-coded metadata and methods for use therewith

ABSTRACT

A video processing device includes a content analyzer that receives a video signal and generates content recognition data based on the video signal, wherein the content recognition data is associated with at least one timestamp included in the video signal. A metadata search device generates time-coded metadata in response to content recognition data and in accordance with the at least one time stamp. A metadata association device generates a processed video signal from the video signal, wherein the processed video signal includes the time-coded metadata.

CROSS REFERENCE TO RELATED PATENTS

The present U.S. Utility Patent Application claims priority pursuant to35 U.S.C. §120 as a continuation of U.S. Utility application Ser. No.14/453,824, entitled “VIDEO PROCESSING DEVICE FOR GENERATING TIME-CODEDMETADATA BASED ON A SEARCH AND METHODS FOR USE THEREWITH”, filed Aug. 7,2014, which is a continuation of U.S. Utility application Ser. No.13/297,471, entitled “VIDEO PROCESSING DEVICE FOR EMBEDDING TIME-CODEDMETADATA AND METHODS FOR USE THEREWITH”, filed Nov. 16, 2011, issued asU.S. Pat. No. 8,842,879 on Sep. 23, 2014, which claims priority pursuantto 35 U.S.C. §119(e) to U.S. Provisional Application No. 61/546,405,entitled “VIDEO PROCESSING DEVICE FOR EMBEDDING TIME-CODED METADATA ANDMETHODS FOR USE THEREWITH”, filed Oct. 12, 2011, all of which are herebyincorporated herein by reference in their entirety and made part of thepresent U.S. Utility patent application for all purposes.

TECHNICAL FIELD OF THE INVENTION

The present invention relates to the processing and display of videosignals.

DESCRIPTION OF RELATED ART

Broadcast television, cable television and direct broadcast satellitenetworks are each mature technologies for providing video content to endusers. In today's world of broadband Internet connections however, usershave a far greater choice as to the source of such content. IPtelevision services are available in most parts of the United Statesthat provide an additional conduit for broadcast video programming. Inaddition, many web services offer streaming services for movies, livesporting events and other video content.

With the proliferation of such sources of video content, users are facedwith an ever expanding array of content and services. Consumption ofmedia is evolving rapidly to the point where the dividing lines start toblur and continuing integration will proceed along multitude ofpathways. However, the fundamental presentation of video programming haschanged very little. Further limitations and disadvantages ofconventional and traditional approaches will become apparent to one ofordinary skill in the art through comparison of such systems with thepresent invention.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 presents a block diagram representation of a video system inaccordance with an embodiment of the present invention.

FIG. 2 presents a block diagram representation of a metadata processingdevice 125 in accordance with an embodiment of the present invention.

FIG. 3 presents a block diagram representation of a video system inaccordance with an embodiment of the present invention.

FIG. 4 presents a pictorial process flow representation in accordancewith an embodiment of the present invention.

FIG. 5 presents a block diagram representation of a video decodingdevice 225 in accordance with an embodiment of the present invention.

FIG. 6 presents a block diagram representation of a video decodingdevice 225 in accordance with an embodiment of the present invention.

FIG. 7 presents a block diagram representation of a video system inaccordance with an embodiment of the present invention.

FIG. 8 presents a block diagram representation of a screen display of avideo display device 20 in accordance with an embodiment of the presentinvention.

FIG. 9 presents a block diagram representation of a screen display of avideo display device 20 in accordance with an embodiment of the presentinvention.

FIG. 10 presents a block diagram representation of a screen display of avideo display device 20 in accordance with an embodiment of the presentinvention.

FIG. 11 presents a block diagram representation of a video system inaccordance with an embodiment of the present invention.

FIG. 12 presents a block diagram representation of a video displaydevice 20 and display device 22 in accordance with an embodiment of thepresent invention.

FIG. 13 presents a block diagram representation of a metadata processingdevice 125 in accordance with another embodiment of the presentinvention.

FIG. 14 presents a block diagram representation of a screen display of apersonal computer 24 in accordance with an embodiment of the presentinvention.

FIG. 15 presents a block diagram representation of a screen display of apersonal computer 24 in accordance with an embodiment of the presentinvention.

FIG. 16 presents a block diagram representation of a screen display of apersonal computer 24 in accordance with an embodiment of the presentinvention.

FIG. 17 presents a block diagram representation of a screen display of apersonal computer 24 in accordance with an embodiment of the presentinvention.

FIG. 18 presents a block diagram representation of a metadata processingdevice 125 in accordance with another embodiment of the presentinvention.

FIG. 19 presents a block diagram representation of a metadata processingdevice 125 in accordance with another embodiment of the presentinvention.

FIG. 20 presents a block diagram representation of a video decodingdevice 225 in accordance with an embodiment of the present invention.

FIG. 21 presents a block diagram representation of a video distributionsystem 375 in accordance with an embodiment of the present invention.

FIG. 22 presents a block diagram representation of a personal computer320 in accordance with an embodiment of the present invention.

FIG. 23 presents a block diagram representation of a video displaydevice 322 in accordance with an embodiment of the present invention.

FIG. 24 presents a block diagram representation of a video storagesystem 179 in accordance with an embodiment of the present invention.

FIG. 25 presents a flow diagram representation of a method in accordancewith an embodiment of the present invention.

FIG. 26 presents a flow diagram representation of a method in accordancewith an embodiment of the present invention.

FIG. 27 presents a flow diagram representation of a method in accordancewith an embodiment of the present invention.

FIG. 28 presents a flow diagram representation of a method in accordancewith an embodiment of the present invention.

FIG. 29 presents a block diagram representation of a screen display of avideo display device 20 in accordance with an embodiment of the presentinvention.

FIG. 30 presents a flow diagram representation of a method in accordancewith an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION INCLUDING THE PRESENTLY PREFERREDEMBODIMENTS

Media such as movies, music, TV shows, sports, user-generated video cancome from many sources. Examples of media sources include satellite,cable, websites, proprietary communication lines and a user's owndevices and local storage. While media flow continues to expand todifferent sources and different devices, metadata, e.g. information,relevant to that media, is also available from different sources such aswebsites, and online encyclopaedias, online or local private databases,and other sources.

FIG. 1 presents a block diagram representation of a video system inaccordance with an embodiment of the present invention. In particular, ametadata processing device 125 is presented that operates to embedmetadata from a metadata source 208 in a video signal 110 to form aprocessed video signal 112. Metadata processing device 125 has theability to integrate relevant metadata to content being processed tocreate a seamless experience that allows a user to enhance their contentconsumption experience. In addition to standard information regardingthe content of video signal, such as title, type of programming, generalinformation, creation time, etc, metadata processing device 125 can usevideo, image and audio processing tools to embed new time-coded metadatawithin the content.

In an embodiment of the present invention, the metadata processingdevice 125 is part of a broadcast video distribution network, such as acable television network, a direct broadcast satellite network, an overthe air television network, an IP television service delivered via abroadband internet connection, a video on demand service, a web-basedvideo service or other video distribution network. Video signals 110 arecombined with metadata from one or more metadata sources as part of theretrieval, distribution and/or transcoding of these video signals toform processed video signals 112 that are sent to a user for viewing.The metadata processing device 125 can be included in a network node,headend, or other network device of a public or private videodistribution network or part of an international, national or localcloud that distributes video as video data as part of a more generallocal area or wide area data network.

Video signal 110 can include a digital video signal complying with adigital video codec standard such as H.264, MPEG-4 Part 10 AdvancedVideo Coding (AVC) including an SVC signal, an encoded stereoscopicvideo signal having a base layer that includes a 2D compatible baselayer and an enhancement layer generated by processing in accordancewith an MVC extension of MPEG-4 AVC, or another digital format such as aMotion Picture Experts Group (MPEG) format (such as MPEG1, MPEG2 orMPEG4), Quicktime format, Real Media format, Windows Media Video (WMV)or Audio Video Interleave (AVI), video coding one (VC-1), VP8, or otherdigital video format. In addition to the inclusion of metadata,processed video signal 112 can include a digital video signal in any ofthe formats discussed above in conjunction with video signal 110. Inparticular, processed video signal 112 can be in the same digital videoformat as video signal 110. As discussed above, in the alternative,processed video signal 112 can be optionally transcoded into a differentcompression format, different video resolution, a different frame rate,a different scale, a different data rate or otherwise into a differentformat, with or without optional scrambling or encryption.

Metadata processing device 125 can mine alternative information sourcesfor new information pertaining to a video signal 110. Such newinformation could be a link to new content or actual new content itself.For example, metadata processing device 125 can include a speechrecognition module that generates a time-coded dialog text of the audioassociated with the video signal 110 and perform Internet searches forhistorical quotes, images, background information and other potentiallyrelevant information. Once new information/metadata is identified it canbe filtered by relevance based on suitability criteria and inserted in atime-coded fashion to the original or transcoded content. This newcontent now contains original content but with relevant metadata thatallows the end user, for example, to understand a video in new ways.

The metadata can be processed by the end user external to the video, butto be compatible with legacy products, the metadata can be watermarkedand embedded in time-coded locations relevant to the content so thatparticular content can also be recompressed with new pictorial oraudible data into a single stream that is viewable by such legacydevices as a single movie the way they understand them today. Multipleversions of content from original to heavily enhanced can be created andmade available to user to choose which version to view. On more advancedviewing devices, the experience can be made to be more enhanced wheremetadata can be selectively rendered or viewed at user discretion on thefinal device. This solution can work for broadcasters, for internetclouds, even for personal clouds in the home when there are personaldata that is not meant for public. For example, family vacation picturesin New Zealand could be added to a viewing a motion picture set in thiscountry, if the user feels it will impress his guest.

Several examples and many optional functions and features includingdetails regarding the inclusion of metadata in a processed video signaland the decoding and presentation of such a processed video signal to auser are presented in conjunction with FIGS. 2-30.

FIG. 2 presents a block diagram representation of a metadata processingdevice 125 in accordance with an embodiment of the present invention. Inparticular, metadata processing device 125 includes a content analyzer200, a metadata search device 204, and a metadata association device 206that is optionally part of a transcoder 190.

In operation, content analyzer 200 receives a video signal 110 andgenerates content recognition data 202 based on the video signal 110.The content recognition data 202 is associated with at least onetimestamp included in the video signal 110. Metadata search device 204generates metadata 205 in response to the content recognition data 202that is time-coded in accordance with the at least one time stamp of thevideo signal 110. Metadata association device 206 generates processedvideo signal 112 from either the video signal 110 or a transcodedversion of video signal 110 generated by optional transcoder 190. Inparticular, the processed video signal 112 includes the time-codedmetadata 205 along with the original or transcoded video signal 110.

The content analyzer 200, metadata search device 204, metadataassociation device 206 and optional transcoder 190 can each beimplemented using a single processing device or a plurality ofprocessing devices. Such a processing device may be a microprocessor,co-processors, a micro-controller, digital signal processor,microcomputer, central processing unit, field programmable gate array,programmable logic device, state machine, logic circuitry, analogcircuitry, digital circuitry, and/or any device that manipulates signals(analog and/or digital) based on operational instructions that arestored in a memory. These memories may each be a single memory device ora plurality of memory devices. Such a memory device can include a harddisk drive or other disk drive, read-only memory, random access memory,volatile memory, non-volatile memory, static memory, dynamic memory,flash memory, cache memory, and/or any device that stores digitalinformation. Note that when content analyzer 200, metadata search device204, metadata association device 206 and optional transcoder 190implement one or more of their functions via a state machine, analogcircuitry, digital circuitry, and/or logic circuitry, the memory storingthe corresponding operational instructions may be embedded within, orexternal to, the circuitry comprising the state machine, analogcircuitry, digital circuitry, and/or logic circuitry.

Content analyzer 200 operates to generate content recognition data 202in a form or format that can be used by metadata search device 204 tosearch one or more metadata sources 208 for metadata 205 to be embeddedin the video. In particular, the content analyzer 200 identifies contentthat occurs at certain points in the video signal 110 based on timestamps included in the video so that metadata associated with thatcontent can be synchronized with the video for presentation to the user.

In an embodiment of the present invention, the content analyzer 200includes a pattern recognition module that uses speech recognition andor image recognition to generate the content recognition data 202 basedon a recognition of speech in audio information included in the videosignal 110 and/or based on image recognition of the particular imagesincluded in the video signal 110. Consider an example where a segment ofvideo at a particular time stamp or range of time stamps shows anautomobile driving along a country road. The audio portion of the videodiscusses the beauty of the Northern Michigan at that time of the year.The pattern recognition module of content analyzer 200 analyzes theimages included in this video segment and recognizes a particularobject, an automobile. In addition, the pattern recognition module ofcontent analyzer 200 analyzes the audio included in this video segmentand recognizes a particular place, Northern Michigan. In response, thecontent analyzer 200 generates content recognition data 202 thatindicates the keywords, “automobile” and “Northern Michigan” associatedwith the timestamp or range of time stamps that are associated with thisparticular portion of video signal 110.

While the content analyzer 200 is described above in speech and imagerecognition, other portions of video signal 110 can be used to generatemetadata associated with the video content. In particular, contentanalyzer 200 can identify content recognition data 202 such as key wordsor other indicators based on closed captioning text included in thevideo signal 110, character recognition of images in the video signal110 and via other identification or recognition routines.

Metadata search device 204 is coupled to one or more metadata sources208 such as local storage, a local area network or a wide area networksuch as the Internet. In an embodiment of the present invention, themetadata search device 204 includes a search engine that searches themetadata source or sources along with a content evaluator that evaluatesthe relevancy of content that was located to identify metadata 205 forinclusion in the processed video signal 112, based on the contentrecognition data 202. In this fashion, content relating to persons,places, objections, quotes, movies, songs, events, or other items ofinterest can be identified for inclusion as metadata 205 in processedvideo 112.

Consider the example discussed above, where a segment of video at aparticular time stamp or range of time stamps shows an automobiledriving along a country road. The key words “automobile” and “NorthernMichigan” indicated by content recognition data 202 are input to asearch engine that, for example, locates web content associated withthese keywords. The web content is evaluated for relevancy based on, forexample its age, image quality, website reviews or other rankings, orother evaluation criteria to determine the particular metadata 205 to begenerated. When the metadata search device 204 generates a plurality ofsearch results, it also generates associated relevance data and selectsthe time-coded metadata 205 based on an analysis of this relevance data.For example, the metadata search device 204 can select the time-codedmetadata 205 by comparing the associated relevance data to a relevancethreshold, by selecting content with the highest relevance, or by otheranalysis of the relevance data or other data associated with theidentified content, such as media format, file size, etc.

In an embodiment of the present invention, the metadata 205 includes theparticular content, the text data, image data, video data and/or audiodata or other media data identified by metadata search device 204. In analternative embodiment, metadata 205 includes links to some or all ofthe identified content in the form of a file address, network addresssuch as a Universal Resource Locator (URL) or other locator, rather thanincluding all of the identified content itself.

The metadata association device 206 generates the processed video signal112 by combining the time-coded metadata with the video signal attime-coded locations in accordance with the at least one time stamp.This can be accomplished in several ways.

In one mode of operation where the metadata 205 includes media content,the processed video signal 112 can be presented as a standard videosignal where metadata in the form of text, images, video are combinedwith the video signal 110 or the transcoded video signal 110 in afashion to be presented in a picture-in-picture, split screen oroverlaid on the original video.

For example, the original video programming from video signal 110 can bepresent in a letterbox or pillar box format with the normally unusedletterbox or pillar box areas filled in with media from metadata 205.Likewise, in a picture-in-picture or split screen mode of operation themedia content from metadata 205 can be presented in a separate portionof the screen from the video programming from video signal 110. Inanother example where the metadata is primarily text or simple images,the metadata 205 can be overlaid on the video programming from videosignal 110. In each of these examples, the processed video signal 112can be formatted for decoding and/or direct display on a legacy videodevice such as a set top box, wireless telephone, personal video player,standard television, monitor or other video display device.

As discussed above, the metadata 205 time-coded based on the time stampsassociated with the content recognition data 202. Metadata 205 caninclude similar time stamps, or ranges of time stamps or other timecoding data that are used to align and synchronize the presentation ofthe metadata 205 with the corresponding portions of the video signal110. In this fashion, portions of the original video corresponding tothe time stamp or range of time stamps that yielded the contentrecognition data 202, are presented contemporaneously with the metadata205 identified by metadata search device 204 in response to thatparticular content recognition data 202. In the mode of operationdiscussed above where the metadata 205 is directly combined with thevideo programming from video signal 110, the metadata association module206 uses the time-coding of metadata 205 to align and synchronize thepresentation of the metadata 205 with the corresponding portions of thevideo signal 110.

In another mode of operation, the metadata association device 206generates the processed video signal 112 by embedding the time-codedmetadata 205 as a watermark on the video signal. In this fashion, thetime-coded metadata 205 in the form of media or media links can bewatermarked and embedded in time-coded locations relevant to the contentso that the video program can also be re-encoded into a single stream.The original video content can be decoded and viewed by legacydevices—however, the watermarking can be extracted and processed toextract either the additional media content or links to additionalcontent that can be viewed with enhanced viewing devices or additionaldisplay devices.

It should be noted that other techniques can be used by the metadataassociation device 206 to combine the content from video signal 110 intothe processed video signal 112. In another mode of operation, thecontent of video signal 110 in the form of video packets can beencapsulated into another protocol that carries the metadata 205. Themetadata 205 and video signal 110 can be extracted by a decoding deviceby unwrapping the outer protocol and passing the video packets to avideo coder for separate decoding. Other techniques includeinterspersing or interleaving the metadata 205 with the video contentfrom video signal 110, transmitting the metadata 205 in a separate layersuch as an enhanced layer of an MVC formatted or other multi-layerformatted video, or transmitting the metadata 205 concurrently with thevideo content of video signal 110 via other time division multiplexing,frequency division multiplexing, code division multiplexing or othermultiplexing technique.

It should also be noted that processed video signal 112 can be presentedin a variety of other formats. A multiplexed audio/video (AV) signalwith digital metadata 205 can be combined in each data packet where theaudio, video and metadata are separated digitally. The metadata 205 canrendered and mixed with the audio or mixed with the video or both andthen reencoded digitally so the metadata is not separable from the audioor video or both. The AV and metadata can be formatted as separatesignals sent out in parallel as distinct signals over distinct paths orthe same path. Also, the AV can be sent contiguously while metadata 205are kept in the metadata processing device 125 (within a local database)for retrieval on demand as required by the final viewing device.

FIG. 3 presents a pictorial process flow representation in accordancewith an embodiment of the present invention. In particular, an exampleprocess flow is shown in conjunction with one particular mode ofoperation of metadata processing device 125.

In the example shown, a segment 130 of video 110 at a particular timestamp shows an automobile driving along a country road. The audioportion of the video discusses the beauty of the Northern Michigan atthat time of the year. The pattern recognition module of contentanalyzer 200 analyzes the images included in this video segment andrecognizes a particular object, an automobile. In addition, the patternrecognition module of content analyzer 200 analyzes the audio includedin this video segment and recognizes a particular place, “NorthernMichigan”. In response, the content analyzer 200 generates contentrecognition data 202 that indicates the keywords, “automobile” and“Northern Michigan” associated with the timestamp or range of timestamps that are associated with this particular segment 130.

The key words “automobile” and “Northern Michigan” indicated by contentrecognition data 202 are input via metadata search device 204 to asearch engine that, for example, locates web content associated withthese keywords. The web content is evaluated for relevancy based on, forexample its age, image quality, website reviews or other rankings, orother suitability criteria to determine the particular metadata 205 tobe generated. When the metadata search device 204 generates a pluralityof search results, it also generates associated relevance data andselects the time-coded metadata 205 based on an analysis of thisrelevance data. In the example shown, Metadata #1 is a portion ofhypertext generated in response to the keywords “Northern Michigan” thatdiscusses bed and breakfast inns Metadata #2 is a portion of hypertextgenerated in response to the keyword “automobile” that includes anadvertisement for a particular model of automobile, the P3000.

As shown in a rendering of the processed video signal 112, segment 130of video 110 is presented in a pillar box format with the pillar boxareas filled in with media from metadata#1 and metadata#2. As discussedin conjunction with FIG. 2, in this mode of operation the processedvideo signal 112 is formatted for decoding and/or direct display on alegacy video device such as a set top box, wireless telephone, personalvideo player, standard television, monitor or other video displaydevice.

FIG. 4 presents a block diagram representation of a video system inaccordance with an embodiment of the present invention. In particular asystem is shown that includes a video decoding device 225 and a videodisplay device 104. In this embodiment, the processed video signal 112is formatted as a standard video signal where metadata 205 in the formof text, images, video are combined with the video signal 110 or thetranscoded video signal 110 in a fashion to be presented in apicture-in-picture, split screen or overlaid on the original video. Inparticular, the processed video signal 112 can be a broadcast videosignal, such as a television signal, high definition television signal,enhanced definition television signal or other broadcast video signalthat has been transmitted over a wireless medium, either directly orthrough one or more satellites or other relay stations or through acable network, optical network or other transmission network. Inaddition, the processed video signal 112 can be generated from a storedvideo file, played back from a recording medium such as a magnetic tape,magnetic disk or optical disk, and can include a streaming video signalthat is transmitted over a public or private network such as a localarea network, wide area network, metropolitan area network or theInternet.

In an embodiment of the present invention, the video decoding device 225includes television receiver, cable television receiver, satellitebroadcast receiver, broadband modem, 3G or 4G transceiver or otherinformation receiver or transceiver that is capable of receiving aprocessed video signal 112 and generating a video/metadata output 118 inresponse thereto. Video decoding device 225 includes a video decoder todecode or transcode the video signal for storage, editing, and/orplayback in a format corresponding to video display device 104. Thevideo decoding device 225 can be implemented via a standard set-top boxand a video display device 104, such as a standard television, monitor,or other video display device. In this implementation, thevideo/metadata output 118 is an S-video signal, component video signal,composite video signal, HDMI signal or other video signal outputcompatible with video display device 104. While shown as separatedevices, the standard format of processed video signal 112 also lendsitself to implementations where the video decoding device 225 and videodisplay device 104 are included in a single device such as a television,computer, tablet, wireless phone or other video display device.

As discussed in conjunction with FIG. 2, processed video signal 112 caninclude the original video programming from video signal 110 in aletterbox or pillar box format with the normally unused letterbox orpillar box areas filled in with media from metadata 205. Likewise, in apicture-in-picture or split screen mode of operation, the media contentfrom the metadata can be presented in a separate portion of the screenfrom the video programming from video signal 110. In another examplewhere the metadata is primarily text or simple images, the metadata canbe overlaid on the video programming from video signal 110. In each ofthese examples, the processed video signal 112 can be formatted fordecoding and/or direct display via one or more legacy video devices suchas a set top box, wireless telephone, personal video player, standardtelevision, monitor or other video display device.

FIG. 5 presents a block diagram representation of a video decodingdevice 225 in accordance with an embodiment of the present invention. Inparticular, a video decoding device 225 is presented for use incircumstances where processed video signal 112 includes embeddedmetadata that requires extraction prior to being displayed. For example,the processed video signal 112 can include time-coded metadata 205embedded as a watermark on the video signal, included as an additionallayer, carried via a protocol that carries the original or transcodedvideo signal in an encapsulated fashion, or that is multiplexed with theoriginal or transcoded video signal. In this implementation, themetadata 205 in the form of additional media content, such as text,images, audio or video, can be extracted and processed for viewing oneither a standard display device or on a separate device from thedisplay device used for displaying the original video content. In thisimplementation, video decoding device 225 includes a metadata extractiondevice 220 that extracts metadata 205 and a video signal 118 from theprocessed video signal 112.

Where the metadata 205 is carried via a watermark, the metadataextraction device 220 processes the processed video signal 112 to readthe watermark and to reconstruct the metadata 205 from the watermark asread. The metadata extraction device 220 optionally removes thewatermark from the processed video signal 112 to generate the video data118 or otherwise passes the processed video signal 112 as the video data118 when removal of the watermark is not necessary for accurate videodecoding.

In circumstances where the metadata 205 is carried via a protocol thatwraps the original or transcoded video content, the metadata 205 andvideo signal 118 can be extracted by a decoding device by unwrapping theouter protocol and passing the video packets to a video coder forseparate decoding. In other metadata transport mechanisms, the metadataextraction device operates via a deinterleaver, demultiplexer or layeredvideo decoder to separate the time-coded metadata 205 form the videodata 118.

A metadata display generator 222 generates metadata display data 116 inresponse to the metadata 205. In particular, the metadata displaygenerator 222 processes the metadata display data 116 for display as thetext, images, video, audio or other media according to the format of themetadata 205 itself Video decoder 224 decodes the video signal 118 togenerate a decoded video signal 114. An output interface 228 generates avideo/metadata output 118 by synchronizing the metadata display data 116to the decoded video signal 114 in accordance with the time stamps inthe decoded video signal and the time-coding of the extracted metadata205.

As shown, the output interface 228 responds to user selections 226 toset the format of the video/metadata output 118. In response to a firstuser selection, the output interface 228 combines the metadata displaydata 116 and the decoded video signal 114 for display of thevideo/metadata output 118 on a single video display device. In thisembodiment, the video/metadata output 118 is formatted as a standardvideo signal where metadata display data 116 in the form of text, imagesor video are combined with the decoded video signal 114 in a fashion tobe presented in a picture-in-picture, split screen or overlaid on theoriginal video, as selected in response to the user selection 226. Inparticular, the video/metadata output 118 can formatted as an S-videosignal, component video signal, composite video signal, HDMI signal orother video signal output compatible with a video display device, suchas video display device 104.

In another mode of operation, the output interface 228 responds to asecond user selection to generate first video/metadata output 118 thatformats the decoded video signal 114 for display on the first videodisplay device such as a video display device 104 and a separate secondvideo/metadata output 118 that includes the metadata display data 116for display on a second display device, such as wireless telephone, atablet personal computer or other personal computer or other display. Inthis fashion, a user can view the video content of decoded video signal114 while contemporaneously viewing the metadata display of thetime-coded metadata, synchronized with the display of the decoded videosignal on a second device.

In a further mode of operation, the user selection 226 includesconfiguration data to configure the output interface 228 to generate thevideo/metadata output 118 to include and exclude selected portions ofthe metadata display data 116 based on a user selection 226. Forexample, the user selection 226 can specify metadata display data 116 tobe included or excluded based on the format of the metadata, such astext, image, audio, video, etc. In this fashion, a user can select thedisplay of text only, text and images, video only, audio only or anyother combination thereof. In addition, the user can provide selectiondata in the form of user selection 226 to select the format of thevideo/metadata output 118 for a plurality of devices including thedigital format including the frame rate, scale and resolution for outputof the decoded video 114, the type of device, format and other/orparameters for providing the metadata display data 116 to one or moreseparate devices via one or more separate video/metadata outputs 118.

FIG. 6 presents a block diagram representation of a video decodingdevice 225 in accordance with an embodiment of the present invention. Inparticular, a video decoding device 225 is presented that includes manysimilar elements described in conjunction with FIG. 5 that are referredto by common reference numerals. Like the video decoding device of FIG.5, this device is for use in circumstances where processed video signal112 includes embedded metadata that requires extraction prior to beingdisplayed. For example, the processed video signal 112 can includetime-coded metadata 205 embedded as a watermark on the video signal,included as an additional layer, carried via a protocol that carries theoriginal or transcoded video signal in an encapsulated fashion, or thatis multiplexed with the original or transcoded video signal.

In this implementation however, the metadata 205 is in the form of linksto additional content. The links are extracted from the processed videosignal 112 via metadata extraction device 220 and the additional contentis retrieved and processed for viewing on either a standard displaydevice or on a separate device from the display device used fordisplaying the original video content. In particular, a metadataretrieval device 234 receives metadata 205 that includes one or morelinks to additional media, such as text, images, audio and video.Metadata retrieval device 234 includes a browser, file retrieval utilityor other hardware or software to generate retrieved metadata 226 byretrieving the media corresponding to these links from one or moremetadata sources 208. In this embodiment, the metadata display generator222 generates the metadata display data 116 to include the media.

The metadata extraction device 220, metadata display generator 222,video decoder 224, output interface 228 and metadata retrieval device234 can each be implemented using a single processing device or aplurality of processing devices. Such a processing device may be amicroprocessor, co-processors, a micro-controller, digital signalprocessor, microcomputer, central processing unit, field programmablegate array, programmable logic device, state machine, logic circuitry,analog circuitry, digital circuitry, and/or any device that manipulatessignals (analog and/or digital) based on operational instructions thatare stored in a memory. These memories may each be a single memorydevice or a plurality of memory devices. Such a memory device caninclude a hard disk drive or other disk drive, read-only memory, randomaccess memory, volatile memory, non-volatile memory, static memory,dynamic memory, flash memory, cache memory, and/or any device thatstores digital information. Note that when metadata extraction device220, metadata display generator 222, video decoder 224, output interface228 and metadata retrieval device 234 implement one or more of theirfunctions via a state machine, analog circuitry, digital circuitry,and/or logic circuitry, the memory storing the corresponding operationalinstructions may be embedded within, or external to, the circuitrycomprising the state machine, analog circuitry, digital circuitry,and/or logic circuitry.

FIG. 7 presents a block diagram representation of a video system inaccordance with an embodiment of the present invention. In particular asystem is shown that includes a video decoding device 225 as part of avideo receiving device 230 and a video display device 104. In thisembodiment, the processed video signal 112 is not formatted as astandard video signal and metadata 205 in the form of text, images,video are combined with the video signal 110 or the transcoded videosignal 110 and must be extracted via video decoding device 225 asdescribed in conjunction with either FIG. 5 or 6.

The processed video signal 112 can be a broadcast video signal, such asa television signal, high definition television signal, enhanceddefinition television signal or other broadcast video signal thatincludes embedded metadata 205 and that has been transmitted over awireless medium, either directly or through one or more satellites orother relay stations or through a cable network, optical network orother transmission network. In addition, the processed video signal 112can be generated from a stored video file, played back from a recordingmedium such as a magnetic tape, magnetic disk or optical disk, and caninclude a streaming video signal that is transmitted over a public orprivate network such as a local area network, wide area network,metropolitan area network or the Internet.

In an embodiment of the present invention, the video receiving device230 includes a television receiver, cable television receiver, satellitebroadcast receiver, broadband modem, 3G or 4G transceiver or otherinformation receiver or transceiver that is capable of receiving aprocessed video signal 112 and generating a video/metadata output 118 inresponse thereto. Video receiving device 230 can be implemented via aset-top box or digital video recorder. In this implementation, thevideo/metadata output 118 is an S-video signal, component video signal,composite video signal, HDMI signal or other video signal outputcompatible with video display device 104. While shown as separatedevices, the video receiving device 230 and video display device 104 canbe in a single device such as a television, computer, tablet, wirelessphone or other video display device. Video receiving device furtherincludes a broadband network interface such as a cable modem, 3G or 4Gmodem, a network card, a digital subscriber line, a wireless networkconnection or other broadband network interface to optionally access oneor more metadata sources 208 via the Internet, via a local area networkconnection or via another connection, either wired or wireless.

User selections 226 generated by a remote control device of the videoreceiving device 230 or via other user interface of the video receivingdevice 230 are sent to the video decoding device 225 to control theoutput selections as presented in conjunction with FIGS. 5-6. In anembodiment of the present invention, the video receiving device 230includes a video signal generator for generating one or more menus,pop-up messages or other graphics and text for generating a graphicaluser interface that is displayed on the video display device 104 andthat prompts the user to enter user selections 226. The user selections226 can include default selections, user preferences or specificselections relating to the display of metadata display data 116, priorto receiving metadata 205 or as metadata 205 is received. Furtherexamples are presented in conjunction with FIGS. 8-10 that follow.

FIG. 8 presents a block diagram representation of a screen display of avideo display device 20 in accordance with an embodiment of the presentinvention. In this example, video display device 20 operates as videodisplay device 104 described in conjunction with FIG. 7. In particular,video display device 20 is coupled to a video receiving device 10 thatoperates as video receiving device 230. In particular video receivingdevice 10 receives a processed video signal 112 that includes a videosignal 110. In this example, the segment 130 of video 110 at aparticular time stamp shows an automobile driving along a country road.The audio portion of the video discusses the beauty of the NorthernMichigan at that time of the year. Metadata 205 embedded in theprocessed video signal 112 includes additional information relating tothe particular place, “Northern Michigan”.

The video decoding device 225 of video receiving device 10 extracts thistime coded meta data 205 during the processing of the correspondingsegment 130 and generates video/metadata output 118 that includes apop-up 26 that is overlaid on the display of video segment 130 thatindicates that additional information is available and that theadditional information relates to Northern Michigan, the keywordsincluded in metadata 205. In an embodiment of the present invention, thepop-up 26 is presented for a limited duration, such as 10 seconds orsome longer or shorter period. If the user responds by, for example,pressing a “yes” or “accept” button on a remote control device 8associated with video receiving device 10 in response to the pop-up,this user selection is used by video processing device 225 to eithermodify the video/metadata output 118 to include the display of thecorresponding metadata display data or to gather further informationrelating to the options for display of the metadata.

FIG. 9 presents a block diagram representation of a screen display of avideo display device 20 in accordance with an embodiment of the presentinvention. In particular a screen display 28 of video display device 20is shown in conjunction with a follow-on to the example presented inconjunction with FIG. 8. In particular, in response to a user requestingto display additional information in response to the pop-up 26, thevideo receiving device modifies the video/metadata output 118 to displaya menu in display screen 28 to prompt the user to select the format ofthe metadata to be displayed, e.g. text only, text and images, or textand video. In addition, the user is given the option of selecting theparticular display mode, e.g. split screen, letterbox or pillar box.

As shown in the example, the user selects “text only” and “split screen”via interaction with the video receiving device 10, via remote controldevice 8. In response, the video receiving device generates userselections, such as user selections 226 that are used to modify thevideo/metadata output 118 in response to these selections. FIG. 10presents a block diagram representation of a screen display of a videodisplay device 20 in accordance with an embodiment of the presentinvention that presents the video segment 130 in conjunction in a splitscreen mode in conjunction with metadata display data as shown in screenportion 32.

While the example presented in conjunction with FIGS. 8-10 contemplatesthe selection of the display of metadata, the type of metadata to bedisplayed and the selection of the display mode in real-time, i.e. atthe time the metadata is received, it should be noted that whether ornot to display metadata at all, the type or types of metadata to bedisplayed, the display mode and possibly further display options couldbe selected by the user and stored as preferences, default settingsduring set-up of the video receiving device 10 or otherwise prior to thedisplay of a particular video segment 130 that has associated metadata.

FIG. 11 presents a block diagram representation of a video system inaccordance with an embodiment of the present invention. In particular, avideo receiving device is presented as described in conjunction withFIG. 8. In this example however, as discussed in conjunction with FIG.5, the video decoding device 225 generates first video/metadata output118 that formats the decoded video signal 114 for display on the firstvideo display device such as a video display device 104 and a separatesecond video/metadata output 118 that includes the metadata display data116 for display on a metadata display device 106, such as a wirelesstelephone, a tablet personal computer or other personal computer orother display. In this fashion, a user can view the video content ofdecoded video signal 114 while contemporaneously viewing the metadatadisplay of the time-coded metadata, synchronized with the display of thedecoded video signal on metadata display device 106.

FIG. 12 presents a block diagram representation of a video displaydevice 20 and display device 22 in accordance with an embodiment of thepresent invention. In particular, video receiving device 10, such asvideo receiving device 230, receives a processed video signal 112 thatincludes a video signal 110. In this example, the segment 130 of video110 at a particular time stamp shows an automobile driving along acountry road. The audio portion of the video discusses the beauty of theNorthern Michigan at that time of the year. Metadata 205 embedded in theprocessed video signal 112 includes additional information relating tothe particular place, “Northern Michigan”.

The video receiving device 10 generates separate video/metadata outputs118. In particular, video receiving device 10 formats the decoded videosignal 114 for display on the video display device 20 and a separatesecond video/metadata output that includes the metadata display data 116for display on a tablet personal computer (PC) 22. In particular, videoreceiving device 10 includes a wireless interface such as a Bluetooth or802.11 transceiver for communicating wirelessly with the tablet PC,either directly or indirectly, via a personal area network, piconet orwireless local area network connection. In this fashion, a user can viewthe video content of decoded video signal 114 while contemporaneouslyviewing the metadata display of the time-coded metadata, synchronizedwith the display of the decoded video signal on tablet PC 22.

FIG. 13 presents a block diagram representation of a metadata processingdevice 125 in accordance with another embodiment of the presentinvention. In particular, an alternative implementation of metadataprocessing device 125 is presented that includes a metadata authoringdevice that generates time-coded metadata 207 in response to contentrecognition data and in accordance with at least one time stamp of avideo signal 110. The metadata 207 can be in the same form or format ofthe metadata 205. In this embodiment however, instead of the metadataprocessing device operating automatically to identify metadata to becombined with the video signal 110, the metadata processing deviceoperates on user control via user input data 212 to specifically selectmetadata from one or more metadata sources and to associate the data toparticular segments of the video signal 110 in a time-coded fashion.

This implementation of metadata processing device is applicable toservice providers, video distribution providers or others that operatein the network or in the cloud to manually select metadata 207 forassociation with particular segments or portions of video signal 110.For example, this provides a tool for advertisers to identify and insertads as metadata 207 in processed video signal 112 sent to users. Thissolution also works for personal clouds in the home where a user want tomanually create a custom viewing experience or insert metadata thatincludes personal text data, image data, video data or audio data thatis not publicly available or is otherwise not meant for publicdissemination. For example, family vacation pictures in New Zealandcould be added to a viewing a motion picture set in this country.Processed video signal 112 can be stored on a digital video recorder orother storage medium and the time-coded metadata 207 could be presentedin conjunction with a viewing of the film either on the same device oron a different display device to enhance the viewing of the film for theuser and the user's guests.

In an embodiment of the present invention, the metadata authoring device210 includes a graphical user interface that allows the user to enteruser input data in the form of keywords that are used by the metadataauthoring device 210 to search, either the Internet or locally storedfiles of the user, for the time-coded metadata 207 to be embedded in theprocessed video signal 112, based on these keywords.

In addition, the metadata authoring device 210 can allow the user togenerate the time-coded metadata 207, based on based on text included inthe user input data. In this fashion the user can directly entercomments or other text to be embedded as metadata 207. As before, themetadata association device 206 generates the processed video signal 112by combining the time-coded metadata 207 with the original or transcodedvideo signal 110 at time-coded locations in accordance with time stampsin the video signal 110. In this fashion, the authored metadata can bepresented in a fashion that is synchronized with the display of theoriginal video content.

FIGS. 14-17 present a block diagram representations of screen displaysof a personal computer 24 in accordance with an embodiment of thepresent invention. In this embodiment, the metadata processing device125 is implemented via person computer 24. In particular, personalcomputer 24 operates to review a video signal, and provides a graphicaluser interface to identify, select or enter metadata to be embedded inprocessed video signal that includes the original or transcoded video.Display screen 250 shows a video review area 300 for reviewing a videosignal 110 and a metadata review area 302 for reviewing metadata thathas previously been embedded. In the example shown, metadata review areais blank area since no metadata has been embedded in conjunction withthis particular portion of the video signal 110.

An activator 304 is presented, that can be selected in response to userinput. In response to the selection of the activator 304, new metadatacan be inserted. In an embodiment of the present invention, in responseto the user selection of the activator 304, the metadata authoringdevice of PC 22 automatically selects a closest time stamp in videosignal 110 for use in time-coding the metadata the is selected forinclusion, based on the portion of the video signal being displayed atthe time the activator 304 was selected. In addition, the display screen252 is presented in response to the selection of the activator 304.Display screen 252 allows the user to search for new metadata to beassociated with the portion of the video signal being displayed.

As shown the user is given the option of browsing local files, searchinglocal files based on a keyword search or other directed search,performing an Internet search, etc. Screen display 254 presents anexample of such metadata that has been located by the user after searchor browsing of either local or Internet content. The user is given theoption to select that content for embedding as time coded meta-data.Display screen 256 presents the same portion of the video after thecontent from display screen 254 has been selected as metadata to beembedded in conjunction with the portion of the video displayed. Asshown, the metadata review area 302 is now populated with the selectedmetadata.

FIGS. 18-19 present block diagram representations of a metadataprocessing device 125 in accordance with other embodiments of thepresent invention. These alternative configurations operate in a similarfashion to the metadata processing devices 125 described in conjunctionwith FIG. 2 and FIG. 13. A plurality of metadata 205 or 207 are embeddedin processed video signal 112 in accordance with selection optionsassociated with particular selection data such as demographic data 214,geographic data 216 or other selection criteria, such as the formatscompatible with various devices including different aspect ratios anddevice resolutions, metadata formats, or other selection criteria. Themetadata 205 or 207 is not only time-coded in accordance with at leastone time stamp of the video signal it is further associated with one ormore forms of selection data so that a decoder can automatically selectportions of the metadata for display, based on selection data present atthe decoder.

In one example, video signal 110 corresponds to a live sporting eventand metadata 205 or 207 is generated that corresponds to an alternativeaudio track with a different play-by-play of the sporting eventassociated with different viewer demographics or different geographicalregions. In this fashion, broadcasters can transmit the live sportingevent but have the play-by-play customized at the user's receivingdevice. For example, viewers in Boston or Los Angeles of a BostonCeltics, Los Angeles Lakers game can each hear a local play-by-play ofthe game. Similarly, alternative game play-by-plays can be directed tousers with differing demographics.

In addition, advertisers could regionalize national ads—e.g. a carcompany could produce a national ad, but leave a blank space where alocal overlay can be inserted that refers to the nearest dealer, thismechanism could pick up the local ad on the fly while data is passingthrough or being transcoded and insert the content served by a nearbyprivate database). The metadata processing device 125, in this example,can insert advertising overlays that are sensitive to, time, content,user profiles, location, etc. This allows more accurate, efficient andprofitable advertising delivery by, for example squeezing the pictureduring late night PSA's or simple programming and use the blank space onthe screen to overlay auctioned ads based on the known local andpreviously inserted metadata or insert advertising or logos in the blankbars when letterbox or pillar box video is being displayed.

FIG. 20 presents a block diagram representation of a video decodingdevice 225 in accordance with an embodiment of the present invention. Inparticular, a video decoding device is presented that includes commonelements previously described, for example, in conjunction with FIGS.5-6. In this embodiment however, metadata extraction device 220generates metadata 205 to include only a selected portion of themetadata that was embodied in the processed video 112. In particular,metadata extraction device selects metadata for generation as metadata205 based on the selection data, such as demographic data 314,geographic data 316 and/or other selection data. In this fashion, thevideo decoding device 225 and/or a video receiving device 230 that it iscoupled to can generate and/or store demographic data 314 based on theinput of user preferences, defaults and/or the conduct of the user inmaking purchases, selecting videos for download, etc. In addition, thevideo decoding device 225 and/or a video receiving device 230 that it iscoupled to can generate and/or store geographic data 314 based on theinput of user preferences, defaults and/or based on a global positioningsystem (GPS) device included in video decoding device 225 and/or a videoreceiving device 230 or other location data. Metadata that is receivedand associated with certain selection criteria such as userdemographics, location information or other selection criteria isretrieved and/or used to generate metadata display data 116 only if thelocal selection data matches the selection criteria that is included inthe metadata received via processed video signal 112.

As discussed in conjunction with FIGS. 18-19, the metadata received viaprocessed video signal 112 can include a plurality of advertisements andmetadata extraction device 220 can generate metadata 205 as only aselected portion of the metadata—i.e. to include only selectedadvertisements, based on the selection data at the video decoding device225. In another example, the metadata received in processed video signal112 includes at least one alternative audio track that is selected andused for output in conjunction with the video/metadata output 118 onlywhen the selection criteria presented in conjunction with this metadatamatches the local selection data at the video decoding device 225.

FIG. 21 presents a block diagram representation of a video distributionsystem 375 in accordance with an embodiment of the present invention. Inparticular, processed video signal 112 is transmitted via a transmissionpath 122 to a video receiving device 10. The video receiving device 10includes a video decoding device 225 to decode the processed videosignal 112 for display of the video portion on video display device 12and the metadata portion of personal computer 14. In an embodiment ofthe present invention, video receiving device can be implemented via aset-top box, digital video recorder, router or home gateway.

The transmission path 122 can include a wireless path that operates inaccordance with a wireless local area network protocol such as an 802.11protocol, a WIMAX protocol, a Bluetooth protocol, etc. Further, thetransmission path can include a wired path that operates in accordancewith a wired protocol such as a Universal Serial Bus (USB) protocol, anEthernet protocol, Media over Coax Alliance (MoCA) protocol or otherhigh speed protocol.

FIGS. 22-23 present pictorial diagram representations of various videodevices in accordance with embodiments of the present invention. Inparticular, television or monitor 322 and portable computer 320illustrate other electronic devices that incorporate a video decodingdevice 225 in accordance with one or more features or functions of thepresent invention. While these particular devices are illustrated, thepresent invention can be implemented in any device that is capable ofdecoding and/or transcoding video content in accordance with the methodsand systems described in conjunction with FIGS. 1-30 and the appendedclaims.

FIG. 24 presents a block diagram representation of a video storagesystem 179 in accordance with an embodiment of the present invention. Inparticular, device 11 is a set top box with built-in digital videorecorder functionality, a stand alone digital video recorder, a DVDrecorder/player or other device that stores the processed video signal112. In this configuration, device 11 can include video decoding device125 that operates to decode the video signal 112 and extract themetadata when retrieved from storage to generate a video signal a formatthat is suitable for display by video display device 12 and further tosend metadata display data for display on another display device such aspersonal computer 14. While particular devices are illustrated, videostorage system 179 can include a hard drive, flash memory device,computer, DVD burner, or any other device that is capable of generating,storing, decoding, transcoding and/or displaying the video content ofprocessed video signal 112 in accordance with the methods and systemsdescribed in conjunction with the features and functions of the presentinvention as described herein.

FIG. 25 presents a flow diagram representation of a method in accordancewith an embodiment of the present invention. In particular, a method ispresented for use in conjunction with one or more functions and featuresdescribed in conjunction with FIGS. 1-24. In step 400, contentrecognition data is generated based on a video signal, wherein thecontent recognition data is associated with at least one timestampincluded in the video signal. In step 402, time-coded metadata isgenerated in response to the content recognition data and in accordancewith the at least one time stamp. In step 404, a processed video signalis generated from the video signal, wherein the processed video signalincludes the time-coded metadata.

In an embodiment of the present invention, speech recognition is used togenerate the content recognition data based on the recognition of speechin audio information included in the video signal. Image recognition canalso used to generate the content recognition data based on therecognition of image data included in the video signal.

The time-coded metadata can include a link to media that includes atleast one of: text data, audio data, video data, and image data. Thetime-coded metadata can directly include at least one of: text data,audio data, video data, and image data. The content recognition data canincludes at least one keyword and the metadata search device can searchfor the time-coded metadata based on the at least one keyword. Step 402can include generating a plurality of search results and associatedrelevance data and selecting the time-coded metadata by comparing theassociated relevance data based on a relevance threshold.

The processed video signal can be generated by combining the time-codedmetadata with the video signal at time-coded locations in accordancewith the at least one time stamp. The processed video signal can begenerated by embedding the time-coded metadata as a watermark on thevideo signal.

FIG. 26 presents a flow diagram representation of a method in accordancewith an embodiment of the present invention. In particular, a method ispresented for use in conjunction with one or more functions and featuresdescribed in conjunction with FIGS. 1-25. In step 410, time-codedmetadata is generated in response to user input data and in accordancewith the at least one time stamp of a video signal. In step 412, aprocessed video signal is generated from the video signal, wherein theprocessed video signal includes the time-coded metadata.

The time-coded metadata can be generated via a graphical user interface.The time-coded metadata can include a link to media that includes atleast one of: text data, audio data, video data, and image data. Thetime-coded metadata can directly include at least one of: text data,audio data, video data, and image data. The user input data can includesat least one keyword and a metadata authoring device can search for thetime-coded metadata based on the at least one keyword. Step 410 caninclude searching at least one of: locally stored files and Internetsearch results.

The time-coded metadata can be generated based on text included in theuser input data. The processed video signal can be generated bycombining the time-coded metadata with the video signal at time-codedlocations in accordance with the at least one time stamp. In addition,the processed video signal can be generated by embedding the time-codedmetadata as a watermark on the video signal.

FIG. 27 presents a flow diagram representation of a method in accordancewith an embodiment of the present invention. In particular, a method ispresented for use in conjunction with one or more functions and featuresdescribed in conjunction with FIGS. 1-26. In step 420 begins byextracting, via a metadata extraction device, metadata and a videosignal from a processed video signal, wherein the metadata is time-codedin accordance with at least one time stamp of the video signal. In step422, metadata display data is generated in response to the metadata. Instep 424, the video signal is decoded to generate a decoded videosignal. In step 426, a video/metadata output is generated bysynchronizing the metadata display data to the decoded video signal inaccordance with the at least one time stamp.

Step 426 can include responding to a first user selection to combine themetadata display data and the decoded video signal for display of thevideo/metadata output on a first video display device. Step 426 caninclude responding to a second user selection to generate firstvideo/metadata output that includes the decoded video signal for displayon the first video display device and second video/metadata output thatincludes the metadata display data for display on a second displaydevice. The video/metadata output can include a selected portion of themetadata display data, based on a user selection.

The time-coded metadata can include a link to media that includes atleast one of: text data, audio data, video data, and image data. Thetime-coded metadata can directly include at least one of: text data,audio data, video data, and image data. The processed video signal canbe generated by embedding the time-coded metadata as a watermark on thevideo signal.

FIG. 28 presents a flow diagram representation of a method in accordancewith an embodiment of the present invention. In particular, a method ispresented for use in conjunction with one or more functions and featuresdescribed in conjunction with FIGS. 1-27. In step 430 begins byextracting, via a metadata extraction device, metadata and a videosignal from a processed video signal, wherein the metadata is time-codedin accordance with at least one time stamp of the video signal. In step432 the method determines if the metadata includes a link to media. Ifso, the method proceeds to steps 436 and 438 where the metadata isretrieved based on the link at metadata display data is generated toinclude the media. Otherwise, in step 434, metadata display data isgenerated in response to the metadata. In step 440, the video signal isdecoded to generate a decoded video signal. In step 442, avideo/metadata output is generated by synchronizing the metadata displaydata to the decoded video signal in accordance with the at least onetime stamp.

FIG. 29 presents a flow diagram representation of a method in accordancewith an embodiment of the present invention. In particular, a method ispresented for use in conjunction with one or more functions and featuresdescribed in conjunction with FIGS. 1-21. Step 450 begins by extracting,via a metadata extraction device, metadata and a video signal from aprocessed video signal, wherein the metadata is time-coded in accordancewith at least one time stamp of the video signal. Step 452 includesgenerating, via the metadata extraction device, a selected portion ofthe metadata based on selection data and the metadata. In step 454,metadata display data is generated in response to the selected portionof the metadata. In step 456, the video signal is decoded to generate adecoded video signal. In step 458, a video/metadata output is generatedby synchronizing the metadata display data to the decoded video signalin accordance with the at least one time stamp.

The selection data can includes demographic data of a user and/orgeographic data corresponding to a location of the user. The metadatacan include a plurality of advertisements and the selected portion ofthe metadata can include at least one selected advertisement of theplurality of advertisements. The metadata can include at least onealternative audio track corresponding to the video signal.

The time-coded metadata can include a link to media that includes atleast one of: text data, audio data, video data, and image data. Thetime-coded metadata can directly include at least one of: text data,audio data, video data, and image data. The processed video signal canbe generated by embedding the time-coded metadata as a watermark on thevideo signal.

FIG. 30 presents a flow diagram representation of a method in accordancewith an embodiment of the present invention. In particular, a method ispresented for use in conjunction with one or more functions and featuresdescribed in conjunction with FIGS. 1-29. Step 460 begins by extracting,via a metadata extraction device, metadata and a video signal from aprocessed video signal, wherein the metadata is time-coded in accordancewith at least one time stamp of the video signal. Step 462 includesgenerating, via the metadata extraction device, a selected portion ofthe metadata based on selection data and the metadata. In step 464, themethod determines if the selected portion of metadata includes a link tomedia. If so, the method proceeds to steps 466 and 468 where the mediais retrieved based on the link and the metadata display data isgenerated to include the media. Otherwise, in step 470, metadata displaydata is generated in response to the selected portion of the metadata.In step 472, the video signal is decoded to generate a decoded videosignal. In step 474, a video/metadata output is generated bysynchronizing the metadata display data to the decoded video signal inaccordance with the at least one time stamp.

While particular combinations of various functions and features of thepresent invention have been expressly described herein, othercombinations of these features and functions are possible that are notlimited by the particular examples disclosed herein are expresslyincorporated in within the scope of the present invention.

As one of ordinary skill in the art will appreciate, the term“substantially” or “approximately”, as may be used herein, provides anindustry-accepted tolerance to its corresponding term and/or relativitybetween items. Such an industry-accepted tolerance ranges from less thanone percent to twenty percent and corresponds to, but is not limited to,component values, integrated circuit process variations, temperaturevariations, rise and fall times, and/or thermal noise. Such relativitybetween items ranges from a difference of a few percent to magnitudedifferences. As one of ordinary skill in the art will furtherappreciate, the term “coupled”, as may be used herein, includes directcoupling and indirect coupling via another component, element, circuit,or module where, for indirect coupling, the intervening component,element, circuit, or module does not modify the information of a signalbut may adjust its current level, voltage level, and/or power level. Asone of ordinary skill in the art will also appreciate, inferred coupling(i.e., where one element is coupled to another element by inference)includes direct and indirect coupling between two elements in the samemanner as “coupled”. As one of ordinary skill in the art will furtherappreciate, the term “compares favorably”, as may be used herein,indicates that a comparison between two or more elements, items,signals, etc., provides a desired relationship. For example, when thedesired relationship is that signal 1 has a greater magnitude thansignal 2, a favorable comparison may be achieved when the magnitude ofsignal 1 is greater than that of signal 2 or when the magnitude ofsignal 2 is less than that of signal 1.

As the term module is used in the description of the various embodimentsof the present invention, a module includes a functional block that isimplemented in hardware, such as via a processing device and/or memory,software, and/or firmware that performs one or module functions such asthe processing of an input signal to produce an output signal. As usedherein, a module may contain submodules that themselves are modules.

Thus, there has been described herein an apparatus and method, as wellas several embodiments including a preferred embodiment, forimplementing various video processing devices. Various embodiments ofthe present invention herein-described have features that distinguishthe present invention from the prior art.

It will be apparent to those skilled in the art that the disclosedinvention may be modified in numerous ways and may assume manyembodiments other than the preferred forms specifically set out anddescribed above. Accordingly, it is intended by the appended claims tocover all modifications of the invention which fall within the truespirit and scope of the invention.

What is claimed is:
 1. A video processing device comprising: a contentanalyzer that receives a video signal and generates content recognitiondata based on the video signal, wherein the content recognition data isassociated with at least one timestamp included in the video signal andwherein the content recognition data includes at least one keywordgenerated by recognizing video content in the video signal ascorresponding to the at least one keyword; and a metadata search device,coupled to the content analyzer and at least one external source, thatsearches for information in response to the content recognition data andgenerates time-coded metadata in accordance with the at least onetimestamp that includes the information and that is associated withselection data, wherein the metadata search device searches the at leastone external source, based on the at least one keyword, to identify theinformation.
 2. The video processing device of claim 1 wherein theselection data is associated with the time-coded metadata to facilitateuse by a viewer device to select portions of the time-coded metadata fordisplay to the viewer in accordance with at least one selectioncriterion.
 3. The video processing device of claim 1 further comprising:a metadata association device, coupled to the metadata search device,that generates a transcoded video signal from the video signal, whereinthe transcoded video signal includes the time-coded metadata and theselection data.
 4. The video processing device of claim 3 wherein themetadata association device generates the transcoded video signal bycombining the time-coded metadata with the video signal at time-codedlocations.
 5. The video processing device of claim 1 wherein theselection data includes one of: a viewer profile, viewer demographicdata or viewer geographical data.
 6. The video processing device ofclaim 1 wherein the selection data includes one of: a device resolutionor a device aspect ratio.
 7. The video processing device of claim 1wherein the selection data includes a time of day.
 8. The videoprocessing device of claim 1 wherein the content analyzer uses speechrecognition to generate the content recognition data based on arecognition of speech in audio information included in the video signal.9. The video processing device of claim 1 wherein the content analyzeruses image recognition to generate the content recognition data based ona recognition of image data included in the video signal.
 10. The videoprocessing device of claim 1 wherein the time-coded metadata includes auniversal resource locator to media that includes at least one of: textdata, audio data, video data, and image data.
 11. The video processingdevice of claim 1 wherein the time-coded metadata includes at least oneof: text data, audio data, video data, and image data.
 12. The videoprocessing device of claim 1 wherein the metadata search devicegenerates the information as a plurality of search results andassociated relevance data and selects the time-coded metadata bycomparing the associated relevance data based on a relevance threshold.13. The video processing device of claim 1 wherein the metadata searchdevice generates the information based on an Internet search and whereinthe at least one external source includes at least one server.
 14. Avideo processing device comprising: a content analyzer that receives avideo signal and generates content recognition data based on the videosignal, wherein the content recognition data is associated with at leastone timestamp included in the video signal and wherein the contentrecognition data includes at least one keyword generated by recognizingvideo content in the video signal as corresponding to the at least onekeyword; a metadata search device, coupled to the content analyzer andat least one external source, that searches for information in responseto the content recognition data and generates time-coded metadata inaccordance with the at least one timestamp that includes theinformation, wherein the metadata search device searches the at leastone external source, based on the at least one keyword, to identify theinformation; and a metadata association device that embeds thetime-coded metadata in a processed video signal in accordance withselection data that indicates at least one selection criterion.
 15. Thevideo processing device of claim 14 wherein the selection data includesone of: a viewer profile, viewer demographic data or viewer geographicaldata.
 16. The video processing device of claim 14 wherein the selectiondata includes one of: a device resolution or a device aspect ratio. 17.The video processing device of claim 14 wherein the selection dataincludes a time of day.
 18. A method comprising: generating contentrecognition data based on a video signal, wherein the contentrecognition data is associated with at least one timestamp included inthe video signal and wherein the content recognition data includes atleast one keyword generated by recognizing video content in the videosignal as corresponding to the at least one keyword; searching forinformation from at least one information source in response to thecontent recognition data based on the at least one keyword, andgenerating time-coded metadata in accordance with the at least onetimestamp, the content recognition data and that includes theinformation; and associating the time-coded metadata with selection datathat indicates at least one selection criterion.
 19. The method of claim18 wherein the selection criterion includes one of: a viewer profile,viewer demographic data or viewer geographical data.
 20. The method ofclaim 18 wherein the selection criterion includes a viewer deviceparameter.